How Scoring Works
The ability to score schedule quality, execution performance, risk modeling, use of logic and more is a key feature of Touchstone and allows project teams to determine how best to assess their portfolio or schedule health.
Touchstone calculates a score for an uploaded schedule based on the acceptance criteria metrics that are defined at the project level. The score determines whether the schedule is accepted or rejected.
You can see the name of the acceptance criteria template that was used for scoring on the Projects Configuration tab. If a schedule is not uploaded for the current period, the score column in the Submittals view for this period is blank.
Metric-Based or Record-Based Scoring
You can specify on the Acceptance Criteria template whether you want Touchstone to use metric-based or record-based scoring.
- Average of Metrics: With metric-base scoring, Touchstone calculates the score using an average of the metric scores. Metric-based scoring allows incremental improvements to tasks to be reflected in the overall score. The metric-based method typically produces higher scores, as a single task needs to fail in multiple areas simultaneously in order to receive a very low score.
The calculation is as follows: Overall Score = (Summation of all of the tripwire formula results for each of the activities across all of the metrics) / (the number of tripwire results for the project).
- Record Fails if One Metric Fails: Record-based scoring calculates the score on a per-record basis. A record or activity scores a 0 if it fails one metric. The final score is an average of the per-record scores. This places a premium on absolute compliance with the assigned metrics. The breach of any single metric on a task prevents those tasks from receiving a "passing grade." This is the default setting.
The calculation is as follows: Overall Score = (Summation of all of the tripwire formula results that are equal to 100 for each of the activities across all of the metrics) / (the number of tripwire results for the project).
The "number of tripwire results" in the above calculations takes into account ONLY the number of metrics that have their "Tripwire Formula" tab enabled on the Metrics tab in Deltek Acumen.
Acceptance Criteria
Each project must have an Acceptance Criteria template. The template sets the minimum acceptance criteria score and specifies the metrics that are used to calculate the acceptance criteria score. You can select the acceptance criteria template on the Projects Configuration tab.
The acceptance criteria operate at the project level; that is, each project is associated with an acceptance criteria template that includes the metrics used to evaluate the submittals. When you submit a schedule in the Submittals view, Touchstone uses the acceptance criteria to determine if the schedule is accepted or rejected.
The Minimum Acceptance Score is set on the Acceptance Criteria Template and is a number between 0 and 100. Touchstone compares an uploaded schedule to the acceptance criteria metrics and scores the schedule based on the metric influence selector values. Influence values indicate how one metric counts towards the overall score for a group of metrics. The higher the value, the more impact the metric in question has on the score.
If the score is equal to or greater than the Minimum Acceptance Score, the schedule passes and Touchstone displays Accepted in the Submittals view Status column. If the score is less than the Minimum Acceptance Score, Touchstone displays Rejected in the Submittals Status column and the submitter can view a report of the issues that caused the schedule to fail.
Scoring Complete
As soon as Touchstone finishes scoring the schedule, the system sends an email to the user who uploaded the schedule (the submitter) and copies the project owner. The email includes the status, the name of the schedule, the score, and the number of errors found in the uploaded schedule. It also includes an Excel report of the score as an attachment (See "Scoring Analysis Report" below).
If the submitter doesn't have an email set up in EPM SA, the system only sends the email to the project owner and notes in the email that the submitter's email is missing.
If the project owner doesn't have an email, the email is sent to the submitter and the Touchstone default sender's email address (set in the Touchstone General setting).
Passing Score
The schedule is accepted if Touchstone checks the uploaded schedule against the acceptance criteria and it scores the same or greater than the Minimum Acceptance Score. A message displays in the Submittals view letting you know that the schedule has been accepted. You can click the link in the Score column ( or ) to view a Microsoft Excel report where you can see how the schedule scored for each metric.
Failing Score
If Touchstone checks the uploaded schedule against the acceptance criteria and it scores less than the Minimum Acceptance Score, the schedule is rejected. A message displays letting you know that the schedule failed. You can click the link in the Score column ( or ) to view a Microsoft Excel report where you can see how the schedule scored for each metric. If the Window Close Date has not yet passed, the issues can be corrected and the schedule can be uploaded again.
Unable to Score
If Touchstone is unable to score the schedule, the score is not updated and the system sends an email to the user who uploaded the schedule (the submitter) and copies the project owner. The email includes a list of things that the submitter needs to correct. If the submitter doesn't have an email set up in EPM SA, the system only sends the email to the project owner and notes in the email that the submitter's email is missing. If the project owner doesn't have an email, the email is sent to the submitter and the default sender's email address (set in Touchstone General setting).
The email subject will be, "Touchstone Submittal Scoring Error" and the message will include a description of the error.
- If scoring fails because there are no metrics in the acceptance criteria, check the acceptance criteria template for the project.
- If scoring fails because of an unknown error, any available exception information is included in the email.
Scoring Analysis Report
After a submittal is scored, Touchstone creates an Excel report that includes the scoring information. This report is emailed out with the status email as an attachment. You can also click the hyperlink in the Submittals view Score column to download the report.
The file name format is: Touchstone - <submitted schedule filename> - <submitted date> Report.xlsx and includes the name of the submitted schedule file as well as the date it was submitted. For example, Touchstone - My Submitted Schedule - 06Feb2019 Report.xlsx.
- Name and Description of the project
- Name of the acceptance criteria template
- Window Close Date and date submitted
- File Name of the submitted schedule
- Score
- A header for each metric in the acceptance criteria followed by any activities that failed the metric
- Threshold colors (for example, green shows a passing score and red shows a failing score)